In the field of interactive graphics application (such as computer games), users typically download, or obtain by physical media, the applications and execute them locally on their devices.
However, some providers of interactive graphics application desire the flexibility and control of executing the graphics applications at a server and streaming the rendered graphics from the applications to users across a network for display on the user's device.
The existing state of the art in this form of streaming graphics content, execute applications on a specialist server that provides a CPU, memory, backing store and a Graphics Processing Unit (GPU) which is used to render the output of the application into a Framebuffer of pixels. The resulting pixels are then retrieved and encoded into a traditional video stream (e.g. h.264) and sent to the client.
There are a number of disadvantages with this approach. Firstly, the servers have to be extremely powerful to run compute and graphics intensive applications for many users simultaneously; this results in a high power usage (and thus also cooling costs) which are a significant issue in determining business viability.
Second, existing video standards such as h.264 are inherently ‘lossy’ meaning that they lose image fidelity during encoding. Compression artifacts can be reduced by increasing the bandwidth requirements of the stream, but there is a hard limit on bandwidth into users' premises and a soft limit on the amount of bandwidth out of the datacenter where the servers are co-located. This means these systems have to accept introducing compression artifacts into the content stream to be viable.
Thirdly, the real-time compression of video is a hugely compute intensive process in which the bandwidth requirements of the resultant stream is a function of the amount of compression processing that has been allocated. This adds to the server load and latency of the system.
Fourthly, millions of consumer devices (e.g. tablets, mobiles, and smart TVs) increasingly contain powerful GPUs that are a resource that is largely under utilized when all the applications graphics processing occurs on the server.
Fifthly, display resolutions are increasing rapidly with many devices now offering 2560×1600 pixels and “4k smartTV” (4096 pixel wide displays) arriving soon. A compression system based around pixels (such as h.264) means that to get the fidelity needed for these displays means increasing the bandwidth of the encoded video stream.
Therefore, it would be desirable if an interactive graphics streaming system could be developed where the application is executed on a server and the graphics rendered by a local GPU at the client device.
One such system is described in the article “Low Delay Streaming of Computer Graphics”, P Eisert and P Fechteler, 15th IEEE International Conference on Image Processing, ICIP 2008. However, the method described in this article includes shadowing the server memory at the client. This shadowing is bandwidth intensive and shadowing of the entirety of the graphics data may not even be necessary (for example, the resolution limitations of the particular client device may not support high resolution textures).
The MPEG-4 standard describes the transmission of compressed geometry meshes and textures to a remote device and could be adapted to provide an interactive graphics streaming system. However, to implement MPEG-4 for a standalone interactive application would require modifications to the application. Furthermore, the MPEG-4 standard would result in potential retransmission of graphics data from the server to the client for each new stream resulting in inefficient use of bandwidth between server and client.
Therefore, an interactive graphics streaming system is desired which provides improved use of the bandwidth between the server and client, is adaptable to different client device capabilities, and requires minimal or no reprogramming of the interactive graphics application.
It is an object of the present invention to provide a method and system for interactive graphics streaming which meets the above desires while overcomes the disadvantages of the prior art, or at least provides a useful alternative.